Skip to content

refactor(ai): skills-only AI surface — real Actions instead of 10 fictional tools, retire the two agents - #512

Merged
os-zhuang merged 2 commits into
mainfrom
claude/agent-metadata-positioning-th5hhm
Jul 28, 2026
Merged

refactor(ai): skills-only AI surface — real Actions instead of 10 fictional tools, retire the two agents#512
os-zhuang merged 2 commits into
mainfrom
claude/agent-metadata-positioning-th5hhm

Conversation

@os-zhuang

Copy link
Copy Markdown
Contributor

HotCRM is the corpus that motivated objectstack-ai/objectstack#3820: 6 skills declaring 16 tool references, 10 of which named tools that exist nowhere. The copilot's instructions claimed abilities it did not have — the runtime silently drops an unresolved tool, so the model improvised or failed when asked to use one. Every instance passed objectstack validate and lint cleanly. This is the dogfood fix, and the first app to run on ADR-0109's authoring model.

The fix is mostly deletion

The fictional references were a category error: "analyze the pipeline", "generate email copy", "score this lead" are reasoning, not tools. What genuinely needed a tool was the handful of steps that change state — and those already exist as HotCRM Actions, reachable through the action_<name> tools the runtime materialises from them. That is ADR-0109's default path: no tool records to author.

Skill Was Now
case_triage triage_case priority rubric in instructions; action_escalate_case / action_close_case do the work
email_drafting generate_email_copy, optimize_subject_line, personalize_content, generate_email the model writes the copy; action_send_email sends it
lead_qualification analyze_lead, suggest_next_action BANT reasoning over get_record; action_convert_lead / action_schedule_followup act
revenue_forecasting analyze_pipeline, identify_at_risk, forecast_revenue aggregate_data / query_records / visualize_data + the forecasting method spelled out
live_data, customer_360 already resolved correctly untouched but for prose

Because every state change now goes through the same Action the UI button uses, permissions and audit are identical whether a human clicks or the assistant calls.

case_triage also handed off to response_drafting — a skill that never existed (the original audit's "hand-off to a nonexistent skill", §7 non-goal because it lives in free-form prompt text). Repointed at the real email_drafting while rewriting the instructions.

The two agents are deleted

sales_copilot and service_copilot were unreachable metadata: ADR-0063 §2 closed *.agent.ts to third parties, and objectstack-ai/cloud#904 made the runtime refuse non-platform agent records. Carrying them is exactly the ADR-0078 lie the whole issue is about. The capability lives in the skills, which attach to the platform assistant by surface affinity — so this removes a fiction without removing a feature.

Prose updated to match (src/docs/crm_admin.md "AI copilots" → "AI skills", plus two stale comments naming the copilots).

Verification

Refs objectstack-ai/objectstack#3820 · ADR-0109 · ADR-0063 · ADR-0078

🤖 Generated with Claude Code

https://claude.ai/code/session_01BHjroNkLkajskKbJaidko4


Generated by Claude Code

…tional tools, retire the two agents (objectstack#3820, ADR-0109)

HotCRM was the corpus that motivated objectstack#3820: 6 skills declaring
16 tool references, of which 10 named tools that exist nowhere. The
copilot's instructions therefore claimed abilities it did not have — the
runtime silently drops an unresolved tool, so the model improvised or
failed when asked to use one. Every instance passed `objectstack
validate` and `lint` cleanly.

The fix is mostly deletion, because the references were a category error:
"analyze the pipeline", "generate email copy", "score this lead" are
REASONING, not tools. What actually needed a tool was the handful of
steps that change state — and those already exist as HotCRM Actions,
reachable through the `action_<name>` tools the runtime materialises from
them (ADR-0109's default path: no tool records to author).

- case_triage: `triage_case` → the priority rubric moves into
  instructions; `action_escalate_case` / `action_close_case` do the work.
  Also repoints the hand-off that named `response_drafting`, a skill that
  never existed (the audit's "hand-off to a nonexistent skill"), at the
  real `email_drafting`.
- email_drafting: 4 fictional copy tools → the model writes the copy;
  `action_send_email` sends it, so recipient resolution, permissions and
  audit match the UI path.
- lead_qualification: `analyze_lead` / `suggest_next_action` → BANT
  reasoning in instructions over `get_record`; `action_convert_lead` /
  `action_schedule_followup` act, and conversion requires confirmation.
- revenue_forecasting: 3 fictional analytics tools → `aggregate_data` /
  `query_records` / `visualize_data` with the actual forecasting method
  spelled out in instructions.
- live_data, customer_360: already correct, untouched but for prose.

The two agents (`sales_copilot`, `service_copilot`) are DELETED. ADR-0063
§2 closed `*.agent.ts` to third parties and cloud#904 made the runtime
refuse non-platform agent records, so they were unreachable metadata —
carrying them is the ADR-0078 lie. The capability lives in the skills,
which attach to the platform assistant by surface affinity.

Verified with the merged reference-integrity rule
(objectstack#3885 `validate-ai-tool-references`): 22 tool references, 0
findings — was 16 references with 10 dead. typecheck, `objectstack
validate`, and the 67-test suite pass.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BHjroNkLkajskKbJaidko4
@vercel

vercel Bot commented Jul 28, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
hotcrm-docs Ready Ready Preview, Comment Jul 28, 2026 3:13pm

Request Review

…DR-0011)

Follow-up to the skills rewrite in this PR. The first pass routed state
changes through `action_escalate_case` / `action_close_case` /
`action_send_email` — but the runtime materialises `action_<name>` ONLY
for an Action that opts in (`ai.exposed` + `ai.description`) AND has a
headless path. All three are `type: 'modal'`: they collect a reason /
resolution / final copy from a person, so they are UI-only and no tool is
ever generated. Referencing them would have re-created the bug this PR
fixes, one layer down.

- `convert_lead` and `schedule_followup` are flow-typed with targets, so
  they only needed the opt-in: both now carry `ai.exposed` + an
  LLM-facing description. `convert_lead` keeps its `confirmText`, which
  the runtime reads as approval-required — an agent invocation lands in
  the HITL queue instead of converting a lead unattended, which is what
  we want for an irreversible outcome.
- `case_triage` and `email_drafting` no longer claim to act. They read,
  reason, and hand a ready-to-paste `reason` / `resolution` / draft to
  the user, pointing at the Escalate / Close / Send Email button. The
  instructions say plainly that the assistant cannot press it — a modal
  Action staying human-driven is the right shape for these, not a gap.

Re-verified with the corrected rule (objectstack#3894, which now models
AI exposure): 19 tool references, 0 findings. Both directions hold — the
two opted-in flow Actions resolve, the three modal ones would not.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BHjroNkLkajskKbJaidko4
@os-zhuang
os-zhuang marked this pull request as ready for review July 28, 2026 15:35
@os-zhuang
os-zhuang merged commit fa0370e into main Jul 28, 2026
8 checks passed
yinlianghui pushed a commit that referenced this pull request Jul 31, 2026
#512 deleted `src/agents/` when the AI surface went skills-only, but seven
maintainer docs kept printing `src/agents/*.agent.ts` in their tree
diagrams and registration tables. `code_examples.md` was the worst of it:
after registering a skill it told the author to "add its name to an agent
in `src/agents/*.agent.ts`" — an instruction that cannot be followed.
`src/cubes/` had the same shape, dropped in favour of datasets (ADR-0021,
noted in objectstack.config.ts) and still drawn in two trees.

Nothing checked, because a path in prose is just prose. That is the same
failure mode as #493 one layer out: metadata validation sees a
well-formed string and stops, so the reference rots silently and the next
reader — human or agent — is sent somewhere that does not exist.

- README.md, AGENTS.md, docs/README.md, docs/STATUS.md,
  docs/ARCHITECTURE.md, docs/developers/code_examples.md: drop the agent
  and cube directories; README picks up `src/datasets/` (the layer that
  actually replaced cubes) and its flow count goes 20 → 23; AGENTS.md's
  `*.action.ts` becomes `*.actions.ts`, the real suffix.
- code_examples.md: drop the skill `permissions: [...]` key — SkillSchema
  has no such field and strips it silently (#511) — and fill the gap the
  deletions leave. The example now says which two sources a `tools` name
  can resolve to (platform data tools, or `action_<name>` from an
  `ai.exposed` Action with a headless path), why `defineTool` is not a
  third one, that reasoning belongs in `instructions` (ADR-0109), and
  where the guard lives.
- ARCHITECTURE.md: same note, plus the `case_triage` skill its table had
  been missing.

Adds a repo-tree guard to test/docs-drift.test.ts — every `src/<dir>/` a
maintainer doc names must resolve on disk, with `docs/archive/` excluded
as a historical record. It earned its place immediately by failing on a
stray `no src/agents/` parenthetical in this change's own first pass.

155 tests (+9), typecheck, validate and build pass; the 2 remaining
validate warnings are the pre-existing campaign_enrollment ones.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y9STuduWbrAwcgviziaV4e
yinlianghui pushed a commit that referenced this pull request Jul 31, 2026
…real

The allowlist added in #557 guessed the platform tool set from what the
`@objectstack/mcp@16.1.0` bridge happens to register, and got it wrong in
both directions:

- It omitted `search_knowledge`, so the guard would have FAILED a
  legitimate reference. The tool is real — it is in
  `PLATFORM_PROVIDED_TOOL_NAMES`, and 16.1.0 documented it all along at
  `spec/src/ai/knowledge-source.zod.ts:114` ("Whether `search_knowledge`
  may expose this source to AI agents"). One grep for that name would
  have caught it; the same absent-from-node_modules inference was
  applied to `visualize_data` and came out the other way.
- It reasoned about deliberately excluding `create_record` /
  `update_record` / `delete_record`. Those are MCP-bridge tools and are
  not in the platform registry at all, so there was nothing to exclude.

The list is now transcribed verbatim from `PLATFORM_PROVIDED_TOOL_NAMES`
in `@objectstack/spec@17.0.0-rc.0` — all 30 entries, diffed equal to the
upstream set — carrying instructions to delete the literal and import it
on the 17.0 upgrade. A hand-copied registry is the drift risk this file
exists to catch, one level up; it is a stopgap for 16.1.0, which exposes
no such export.

Cross-checked against upstream's own `ai-skill-tool-unresolved` rule,
which ships in `@objectstack/lint@17.0.0-rc.0` (objectstack#3820). Both
give identical verdicts on seven probes: `search_knowledge`,
`query_data`, `todo_write` and `action_convert_lead` accepted;
`action_escalate_case` (not `ai.exposed`), `search_knowledgebase` and
`triage_case` rejected.

Also corrects the "nothing defines it" claim in three places — the
`customer_360` docstring, its changeset, and `code_examples.md`. The
skill still does not reference the tool, for a narrower and accurate
reason: retrieval needs a declared knowledge source, `AIKnowledgeSchema`
mounts only on `AgentSchema.knowledge`, and #512 deleted the agents — so
a skills-only app has nowhere to declare one and the tool would resolve
and return nothing. The skill's instruction line now says the model has
no such tool *in this skill*, rather than that none exists.

168 tests pass, typecheck and build clean, validate holds at its 2
pre-existing warnings.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y9STuduWbrAwcgviziaV4e
yinlianghui pushed a commit to yinlianghui/hotcrm that referenced this pull request Jul 31, 2026
…in CI (objectstack-ai#493)

`customer_360` declared `tools: ['search_knowledge']` — the survivor of the
eleven fictional tools objectstack-ai#493 counted, missed by objectstack-ai#512 because that skill
looked "already correct". The runtime silently drops an unresolved tool,
so the skill shipped with its only declared capability resolving to
nothing, while its instructions promised an account + cases +
opportunities + knowledge roll-up it had no tool to read.

Defining the missing tool was never the fix: `ToolSchema` is a read-only
Studio projection — no `implementation` field, no executor loads it — so a
hand-authored `search_knowledge` would validate, build, and still never
run. And the knowledge base here is not a search service: it is
`crm_knowledge_article`, a normal object. The skill now reads it with
`query_records` alongside the contacts, cases and opportunities the
description already promised, and quotes totals from `aggregate_data`.

Adds `test/skills-integrity.test.ts` (its own file — metadata-references
is a busy merge surface):

- every skill tool resolves to a platform built-in or an `action_<name>`
  tool materialised from a real Action, with wildcard subscriptions
  resolved by prefix;
- every referenced Action is `ai.exposed` with a ≥40-char LLM description
  AND a headless path (flow with a defined target, or a script body) —
  the regression objectstack-ai#512's follow-up commit had to fix by hand;
- stack tool metadata cannot be used to satisfy a dangling reference,
  since it never executes;
- every skill handed off to in instructions exists — the defect that
  pointed `case_triage` at a `response_drafting` skill that was never
  defined.

Each guard verified against the real defect before landing: reinstating
`search_knowledge`, `action_escalate_case` and the `response_drafting`
hand-off each fails its own assertion.

Also corrects two stale docstrings. `case_triage` and `email_drafting`
justified their read-only posture by calling `escalate_case`,
`close_case` and `send_email` `type: 'modal'` with no headless path;
objectstack-ai#515 and the send_email rewrite retyped all three to flow/script. The
posture is right, the reason was wrong — none of the three opts in via
`ai.exposed` (ADR-0011 is default off), so no tool is materialised and
the human review step stays a deliberate governance call.

typecheck, validate, build and the full suite pass (146 tests, +6).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y9STuduWbrAwcgviziaV4e
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants